Systems and methods for magnetic resonance imaging standardization using deep learning

ABSTRACT

A computer-implemented method for transforming magnetic resonance (MR) imaging across multiple vendors is provided. The method comprises: obtaining a training dataset, wherein the training dataset comprises a paired dataset and an un-paired dataset, and wherein the training dataset comprises image data acquired using two or more MR imaging devices; training a deep network model using the training dataset; obtaining an input MR image; and transforming the input MR image to a target image style using the deep network model.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of International Patent ApplicationNo. PCT/US19/37235, filed Jun. 14, 2019, which claims priority to U.S.Provisional Application No. 62/685,774 filed on Jun. 15, 2018, each ofwhich is incorporated herein in its entirety.

BACKGROUND

A common task for radiologists is to compare sequential imaging studiesacquired on different magnetic resonance (MR) hardware systems. Becauseeach manufacturer's images show different contrast or distortions due todifferent design considerations, this task can be challenging. Clinicalimaging trials can be more challenging if multiple vendor scanners areinvolved. Therefore, it is desirable to transform MR images from theappearance of one vendor to another vendor, or to a standardized MRstyle.

SUMMARY

The present disclosure provides methods and systems are capable oftransforming magnetic resonance (MR) images from the appearance of onevendor to another vendor, or to a standardized MR image form or style.For example, the provided methods and systems may preserve anatomicalinformation while transforming the vendor specific contrast “style”.Methods and systems of the present disclosure may lead to a universal oruniform MRI style which benefits patients by improving inter-subjectreproducibility and accuracy of diagnoses, enabling quantifiablecomparison, consistency, standardization, and allowing MRI to be morequantitative and standardized.

In one aspect of the invention, a computer-implemented method fortransforming magnetic resonance (MR) imaging across multiple vendors maybe provided, said method comprising: obtaining a training dataset,wherein the training dataset comprises a paired dataset and an un-paireddataset, and wherein the training dataset comprises image data acquiredusing two or more MR imaging devices; training a deep network modelusing the training dataset; obtaining an input MR image; andtransforming the input MR image to a target image style using the deepnetwork model.

Additional aspects of the invention may be directed to a non-transitorycomputer-readable storage medium including instructions that, whenexecuted by one or more processors, cause the one or more processors toperform operations comprising: obtaining a training dataset, wherein thetraining dataset comprises a paired dataset and an un-paired dataset,and wherein the training dataset comprises image data acquired using twoor more MR imaging devices; training a deep network model using thetraining dataset; obtaining an input MR image; and transforming theinput MR image to a target image style using the deep network model.

Additional aspects and advantages of the present disclosure will becomereadily apparent to those skilled in this art from the followingdetailed description, wherein only illustrative embodiments of thepresent disclosure are shown and described. As will be realized, thepresent disclosure is capable of other and different embodiments, andits several details are capable of modifications in various obviousrespects, all without departing from the disclosure. Accordingly, thedrawings and descriptions are to be regarded as illustrative in nature,and not as restrictive.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings (also “Figure” and “FIG.” herein) of which:

FIG. 1 schematically illustrates a system for providing standardized MRimages based on deep learning.

FIG. 2 shows an example of cross-vendor transformation.

FIG. 3 shows a result generated using the methods and systems of thepresent disclosure.

FIG. 4 and FIG. 5 show comparison of residual errors of the cross-vendortransformation result.

FIG. 6 illustrates an example of method for transforming MR image dataof one or more styles to image data of a target style.

FIG. 7 shows a block diagram of an example of MR imaging standardizationsystem, in accordance with some embodiments of the present disclosure.

DETAILED DESCRIPTION

While various embodiments of the invention have been shown and describedherein, it will be obvious to those skilled in the art that suchembodiments are provided by way of example only. Numerous variations,changes, and substitutions may occur to those skilled in the art withoutdeparting from the invention. It should be understood that variousalternatives to the embodiments of the invention described herein may beemployed.

Recognized herein is a need for transforming magnetic resonance (MR)images from the appearance of one vendor to another vendor, or to astandardized MR style. Methods and systems of the presenting disclosuremay be capable of transforming MR images taken from different MRscanners into a synthesized or standardized form. The synthesized orstandardized form may be a target form with pre-determinedcharacteristics such as contrast, resolution, image size, color,skewness, distortion, orientation, and the like. Alternatively or inaddition to, the target form may be consistent with the form or style ofimages taken by a selected scanner (provided by a selected vendor). Insome cases, different MR scanners or MR imaging devices may be ofdifferent types provided by the same and/or different vendors. In somecases, different MR scanners or MR imaging devices may have differentsettings or hardware designs/configurations such that the correspondingMR images may be different in at least one of contrast, resolution(e.g., thickness size, pixel size, etc) and image distortion.

As used herein, the terms “style”, “image style” and “form” may refer tothe appearance of an image which can be used interchangeably throughoutthe specification.

Methods and systems disclosed herein may provide improved accuracy ofcross-vendor transforms for contrast-weighted MRI. The provided methodsand systems may use deep learning methods trained with multi-vendordatasets. The provided technique can significantly improve clinicalworkflow of comparing sequential scans taken on different scanners. Thismethod may further improve quantifying biomarkers, co-registration andsegmentation across images collected longitudinally. It is an essentialcomponent pushing MR imaging to be a standardized and quantitativeimaging modality.

Methods disclosed herein may provide an algorithm, by training on bothco-registered paired datasets from the same subject and furtherenhancing with un-paired training using Cycle Generative AdversarialNetwork (Cycle-GAN), results in accurate cross-vendor transformation.The provided methods and systems may be capable of standardizing thecontrast-weighted MRI images into the same contrast standard, whichenable easier longitudinal and cross-sectional analysis and comparison.Such an application is essential and valuable for clinical radiologiststo monitor and staging disease progress. Also with normalized imagecontrasts, this technology can also be used to improve the tasks such asquantifying biomarkers, co-registration and segmentation.

High quality medical image datasets can be rare. Paired andco-registered cross-vendor images from the same subject can be evenharder to collect. The provided method may utilize un-paired trainingapproach allowing the deep learning method to train and apply onexisting larger datasets that are already available in clinicaldatabase.

In addition, method disclosed herein may further allow MRI to be a morestandardized and quantitative imaging modality with betterquantification and repeatability. It is also a complementary technologyto direct parameter mapping techniques (e.g., MRF) and achievestandardized imaging directly from routine MRI sequences. Methods orsystems of the represent disclosure can be applied to other modalitiessuch as positron emission tomography (PET), X-ray, computed tomography(CT) and ultrasound, when image standardization across imaging devicesof different types, setups or configurations is desired.

FIG. 1 schematically illustrates a system 100 using deep learning toprovide standardized MR images. The system 100 may be capable oftraining a network model for cross-vendor transformation andstandardization. The system 100 may train the network model by trainingon both co-registered paired datasets from the same subject and furtherenhancing with un-paired training using Cycle-GAN, results in accuratecross-vendor transformation. The provided methods and systems may becapable of standardizing the contrast-weighted MRI images into the samecontrast standard, which enable easier longitudinal and cross-sectionalanalysis and comparison. The network model may be used to transform MRimage data acquired by one or more MR scanners or MR imaging devices toa target style. The original MR image data and the target image may bedifferent in at least one of the following: contrast, resolution andimage distortion. In some cases, the MR image may be acquired bydifferent MR scanners or MR imaging devices 101, 103 that may be ofdifferent types provided by the same and/or different vendors. In somecases, the different MR scanners or MR imaging devices 101, 103 may havedifferent settings, hardware designs, or configurations such that thecorresponding MR images may be different in at least one of contrast,resolution and image distortion.

Deep Learning Approach

During the image transformation, a deep learning algorithm may beapplied to the original image to estimate a function ƒ that transformsthe original image m_(a) of any form or style to the target image {tildeover (m)}. The target image may conform to a synthesized or standardform. As described above, the standardized form (i.e., target form) mayhave pre-determined characteristics such as contrast, resolution (e.g.,thickness size, pixel size, etc), image size, color, skewness,distortion, orientation, or other characteristics. Alternatively or inaddition to, the target form may be consistent with the form or style ofMR images taken by a selected scanner (provided by a selected vendor).Define {tilde over (m)} as the target image, then the transformationfrom image of any form or style m_(a) to the target image {tilde over(m)} can be formulated as:{tilde over (m)}=ƒ(m _(a)),

where ƒ represents the image transformation from any style to a standardstyle. In some cases, this function ƒ may be obtained by optimizingmetrics g between the ground-truth image m and the estimated image{tilde over (m)} through a training process on a number of trainingdatasets:min Σg _(i)(k(m),({tilde over (m)})),s.t. {tilde over (m)}=ƒ(m _(a))

There can be one or more cost metrics which can be combined withoptimized weightings. g can be any suitable metrics such as l₂ norm∥k(m)−k({tilde over (m)})∥₂, l₁ norm ∥k(m)−k({tilde over (m)})∥₁,structural dissimilarity, structural similarity loss, perceptual loss orother metrics. In some cases, k can be identity transform then themetrics are calculated in the image domain. k can be any othertransforms, such as Fourier transform, therefore the metrics may becalculated in the corresponding frequency domain. In some cases, the gmetric may be used as criteria during the training process of the deeplearning model. In some cases, the g metrics can also be a network modelthat is separately or simultaneously trained together with ƒ, todiscriminate image states and evaluate image quality.

Similarly, the network model may estimate a function that transforms MRimage acquired by one scanner provided by a first vendor to imageshaving a style consistent with a second vendor. Define m₁ as the image111 from the acquisition performed by the first scanner 101, m₂ as theimage 113 having a form or style from the acquisition performed by thesecond scanner 103, then the transformation from m₁ to m₂ can beformulated as:m ₂=ƒ_(1→2)(m ₁),

Similarly, the transformation from m₂ to m₁ can be formulated as:m ₁=ƒ_(2→1)(m ₂),

where ƒ_(1→2) and ƒ_(2→1) represent the corresponding imagetransformation respectively.

The provided methods and systems may train a neural network model fortransforming the MR image data. In some cases, the neural network mayuse U-net neural network structures, which have been widely used formedical tasks such as segmentations and image enhancement. The U-netneural network structures may be used in this network for image-to-imageregression tasks.

The neural network may employ any type of neural network model, such asa feedforward neural network, radial basis function network, recurrentneural network, convolutional neural network, deep residual learningnetwork and the like. In some embodiments, the machine learningalgorithm may comprise a deep learning algorithm such as convolutionalneural network (CNN). Examples of machine learning algorithms mayinclude a support vector machine (SVM), a naïve Bayes classification, arandom forest, a deep learning model such as neural network, or othersupervised learning algorithm or unsupervised learning algorithm. Insome cases, the method may be a supervised deep machine learning method,an unsupervised deep machine learning method or a combination of both.

The deep learning network such as CNN may comprise multiple layers. Forexample, the CNN model may comprise at least an input layer, a number ofhidden layers and an output layer. A CNN model may comprise any totalnumber of layers, and any number of hidden layers. The simplestarchitecture of a neural network starts with an input layer followed bya sequence of intermediate or hidden layers, and ends with output layer.The hidden or intermediate layers may act as learnable featureextractors, while the output layer in this example provides MR imagesconforming to a target style or form.

Each layer of the neural network may comprise a number of neurons (ornodes). A neuron receives input that comes either directly from theinput data (e.g., image data taken on one or more scanners of differenttypes) or the output of other neurons, and performs a specificoperation, e.g., summation. In some cases, a connection from an input toa neuron is associated with a weight (or weighting factor). In somecases, the neuron may sum up the products of all pairs of inputs andtheir associated weights. In some cases, the weighted sum is offset witha bias. In some cases, the output of a neuron may be gated using athreshold or activation function. The activation function may be linearor non-linear. The activation function may be, for example, a rectifiedlinear unit (ReLU) activation function or other functions such assaturating hyperbolic tangent, identity, binary step, logistic, arcTan,softsign, parameteric rectified linear unit, exponential linear unit,softPlus, bent identity, softExponential, Sinusoid, Sinc, Gaussian,sigmoid functions, or any combination thereof.

The deep learning network models may be trained both on paired datasetsand on un-paired datasets which beneficially provides flexibility indata collection. The training datasets may comprise paired datasetsincluding a reference image of a target style and an original image, andun-paired datasets which may include image data acquired by a pluralityof scanners or imaging apparatuses of different types.

In some cases, the paired datasets may be used in supervised training.In some embodiments, the training process of the deep learning model mayemploy a residual learning method. In some instances, the residuallearning framework may be used for evaluating a trained model. In someinstances, the residual learning framework with skip connections maygenerate estimated ground-truth images from the original images such asMR image taken on a given scanner (provided by a given vendor), withrefinement to ensure it is consistent with measurement (dataconsistency). In some cases, what the model learns is the residual ofthe difference between the original image data and ground-truth/targetimage data, which is sparser and less complex to approximate using thenetwork structure. The method may use by-pass connections to enable theresidual learning. In some cases, a residual network may be used and thedirect model output may be the estimated residual/error between theoriginal image and the target image. In other word, the function to belearned by the deep learning framework is a residual function which insome situations may be easy to optimize. The target image can berecovered by adding the original image to the residual. This residualtraining approach may reduce the complexity of training and achievebetter performance where the output level is small.

In some cases, the deep learning model may be trained with adaptivelytuned parameters based on user input and real-time estimated outputimages. Alternatively or in addition to, the deep learning network maybe a “plain” CNN that does not involve residual learning. In some cases,during the training process, the deep learning model may adaptively tunemodel parameters to approximate the reference image of target style froman initial set of the input images, and outputting an image of thetarget style.

In the super-vised training based on MR image data, the cost functionmay need to be able to identify the differences between 3D voxels or 2Dimages. The paired dataset may be pre-processed to reduce thedisplacement or offset of corresponding pixels/voxels of in the paireddata so that the cost function or loss function such as L1 loss (i.e.,mean absolute error), L2 loss (i.e., mean square error), structuralsimilarity loss, or perceptual losses can applied. For example, in orderto compute similarity or differences estimation for super-visedlearning, image/volume co-registration algorithms may be applied togenerate spatially matched images/volumes. In some cases, theco-registration algorithms may comprise a coarse scale rigid algorithmto achieve an initial estimation of an alignment, followed by afine-grain rigid/non-rigid co-registration algorithm. In some cases, thesupervised losses may be pixel-wise L1 and/or L2 losses. Alternativelyor in addition to, the supervised losses may be voxel-wise loss,sub-image-wise losses or others.

In some cases, a network model trained based on paired datasets may befurther enhanced using un-paired datasets. In some cases, a supervisedlearning and unsupervised learning may be performed sequentially. Insome situations unsupervised algorithms may introduce instability duringtraining. To avoid such instability, it is beneficial to train a modelusing supervised training with paired datasets then further enhance themodel using unsupervised learning. For example, the model may beinitially trained to estimate a transformation between differentcontrast styles using supervised losses such as pixel-wise L1 and/or L2losses. The performance of the resulting model may not be good enoughdue to limitation of the supervised losses and the amount of availablepaired dataset. The model may be further improved by unsupervisedlearning or a combination of unsupervised and supervised learning. Forexample, the model can be further refined or enhanced using refinementlosses such as a mixed loss of supervised losses (e.g., L1 loss, L2loss, Lp loss, structural similarity, perceptual losses, etc) andunsupervised losses (e.g., GAN (Generative Adversarial Network) loss,least-square GAN, WGAN losses (Wasserstein GAN), etc).

There may be multiple iterations in a training process. In each of themultiple iterations, different supervised losses, unsupervised losses orcombinations of supervised losses and unsupervised losses may beselected. Below lists examples of loss functions that may be involved inan exemplary training process to optimize the transform network f_(2→1)and f_(1→2), with co-registration ϕ align volume/image m₁ tovolume/image m₂:

${= {f_{1\rightarrow 2}\left( {\phi\left( m_{1} \right)} \right)}}{= {f_{2\rightarrow 1}\left( {\phi^{- 1}\left( m_{2} \right)} \right)}}{{Loss}_{final} = {{Loss}_{supervised} + {Loss}_{unsupervised}}}{{Loss}_{supervised} = {{\Sigma_{{supervised}{loss}p}w_{p}{L_{p}\left( {,m_{1}} \right)}} + {w_{p}^{\prime}{L_{p}\left( {,m_{2}} \right)}}}}{{Loss}_{unsupervised} = {{w_{cycle}{Loss}_{{cycle} - {GAN}}} + {Loss}_{GAN}}}{{Loss}_{{cycle} - {GAN}} = {{L_{p}\left( {{f_{2\rightarrow 1}\left( {f_{1\rightarrow 2}\left( m_{1} \right)} \right)},m_{1}} \right)} + {L_{p}\left( {{f_{1\rightarrow 2}\left( {f_{2\rightarrow 1}\left( m_{2} \right)} \right)},m_{2}} \right)}}}{{Loss}_{GAN} = {{L_{2}\left( {D{()}} \right)} + {L_{2}\left( \left( {{D\left( m_{2} \right)} - 1} \right)^{2} \right)}}}{{\min\limits_{D}{V_{LSGAN}(D)}} = {{\frac{1}{2}{E_{x \sim {{pdata}(m_{2})}}\left\lbrack \left( {{D\left( m_{2} \right)} - 1} \right)^{2} \right\rbrack}} + {\frac{1}{2}{E_{\sim {p{()}}}\left\lbrack \left( {D\left( {G(z)} \right)} \right)^{2} \right\rbrack}}}}{{\min\limits_{G}{V_{LSGAN}(G)}} = {\frac{1}{2}{E_{\sim {p{()}}}\left\lbrack \left( {{D\left( {G(z)} \right)} - 1} \right)^{2} \right\rbrack}}}$

The training process may involve supervised and unsupervised learningtechniques that can be applied sequentially or concurrently. Theun-paired datasets may be used for unsupervised training which enablesthe method to further train and apply on most or all existing largescale MRI datasets. In some cases, the system 100 and/or methods mayemploy Cycle Generative Adversarial Network (Cycle-GAN) that furtherenables improved performance and more flexible training on both paireddatasets and un-paired datasets. A Cycle-GAN may be used in adversarialtraining in which a discriminative network is used to enhance theprimary network. The primary network may be generative (segmentation,synthesis) or discriminative (classification). With the adversarialtraining, the deep learning neural network model may be learnt withcontent and style loss. The content loss may be used to ensure theconsistency of anatomical information over the image transformation. Thecontent loss can be quantified using supervised voxel-wise losses orpixel-wise losses such as L1 loss, L2 loss, structural similarity,perceptual losses or others as described elsewhere herein. The styleloss may be used to ensure the output result preserve the designedcontrast visual quality, which can be estimated using the statistic offeatures from selected networks, such as the histogram of the activationin selected layer of the network. In some cases, the adversarial loss ofthe discriminator can be a type of style loss to estimate the style ofthe image by learning to predict if output result has the desired imagevisual quality. The machine learnt network may further be configured asa U-net. The U-net is an auto-encoder in which the outputs from theencoder-half of the network are concatenated with the mirroredcounterparts in the decoder-half of the network. The U-net may replacepooling operations by upsampling operators thereby increasing theresolution of the output.

In some cases, the training process of the deep learning model mayemploy a patch-based approach. In some cases, the paired datasets may bedivided into patches. For example, a pair of training images such as apair of original image and target image may each be divided spatiallyinto a set of smaller patches. The high quality image and the lowerquality image can be divided into a set of patches. A size of an imagepatch may be dependent on the application such as the possible size arecognizable feature contained in the image. Alternatively, the size ofan image patch may be pre-determined or based on empirical data.

The trained deep learning model may be used for transforming input datacomprising MR image data of any style (e.g., taken on a first scanner)or different styles (e.g., taken on scanners of different types) totarget data having a target style. In some cases, the input data may be3D volume comprising multiple axial slices. In an example, an input andoutput slices may be complex-valued images of the same or differentsize, resolution, contrast or other characteristics. With aid of theprovided system, automated MR image standardization may be achieved.

Example Datasets

In an example, datasets were collected and included co-registeredmulti-vendor MRI datasets from 7 subjects on different 3T scanners (GEMR750, Philips Ingenia, Siemens Skyra). There are co-registered datasetson 3 subjects collected with similar settings using both GE MR750 andPhilips Ingenia, while another 4 subjects collected using both GE MR750and Siemens Skyra. Additionally, there are 25 un-co-registered samplesfrom different subjects that can be used for unpaired training withCycle-GAN to ensure robust training and avoid over-fitting.

In the example, the performance of standardizing commoncontrast-weighted sequences is examined: 1) Axial-2D-T1w, 2)Axial-2D-T2w, 3) Axial-2D-GRE and 4) Sagittal-3D-T1-FLAIR. For eachdataset, there are around 28˜32 slices for 2D images and around 200˜300planes for high resolution 3D images.

Evaluation

Evaluation on contrast standardization results on the series for T1w,T2w, GRE and FLAIR is performed. In an example, the evaluation metricsmay include, but not limited to, Peak-Signal-to-Noise-Ratio (PSNR),normalized Root-Mean-Squared-Error (RMSE) and Structural SimilarityIndex (SSIM). The real acquired images on different scanners arecompared with the results of cross-vendor transforms.

Results

FIG. 2 shows an example of cross-vendor transformation 200. As shown inthe example, accurate cross-vendor transformation (T1w, vendor #2 tovendor #1 shown as example) is generated by the provided method orsystem. The inter-vendor differences between the two images are reducedafter transformation while preserving the diagnostic quality as well asoriginal anatomical information from the acquired T1w image.

A fast and accurate inference may be achieved using the network modeltrained in a processed as described above. As shown in FIG. 3 , thesimilarity metrics 300 (average statistics for T1w, T2w and GRE)improves significantly (p<0.0001) by using the proposed cross-vendorstandardization: over 5.0 dB PSNR gain, around 30% reduction in RMSE andover 0.15 improvements for SSIM.

FIG. 4 and FIG. 5 further compare the detailed residual errors ofcross-vendor transformation (T2w, GRE, vendor #3 to vendor #1) withzoom-in visualizations of the results and errors.

From the comparison, it is shown that the provided algorithm, bytraining on both co-registered paired datasets from the same subject andfurther enhancing with un-paired training using Cycle-GAN, results inaccurate cross-vendor transformation. The provided system and method maybe capable of standardizing the contrast-weighted MRI images into thesame contrast standard, which enable easier longitudinal andcross-sectional analysis and comparison. Such an application isessential and valuable for clinical radiologists to monitor and stagingdisease progress. Also with normalized image contrasts, this technologycan also be used to improve the tasks such as quantifying biomarkers,co-registration and segmentation.

Systems and methods of the present disclosure may provide an MR imagingstandardization system that can be implemented on any existing MRimaging system without a need of a change of hardware infrastructure.The MR imaging standardization system may be implemented in software,hardware, firmware, embedded hardware, standalone hardware, applicationspecific-hardware, or any combination of these. The MR imagingstandardization system can be a standalone system that is separate fromthe MR imaging system. Alternatively or in addition to, the MR imagingstandardization system can be integral to an existing MR imaging systemsuch as a component of a controller of the MR imaging system.

Methods and systems of the present disclosure can be implemented by wayof one or more algorithms. An algorithm can be implemented by way ofsoftware upon execution by the central processing unit. For example,some embodiments use the algorithm illustrated in FIG. 1 and FIG. 5 orother algorithms provided in the associated descriptions above.

FIG. 6 illustrates an example of method 600 for transforming MR imagedata of one or more styles to image data of a target style. MR imagesmay be obtained from one or more MR imaging devices/scanners (operation610) for training a deep learning model. The one or more scanners may beof different types, different hardware designs, differentconfigurations, or be provided by different vendors. The MR images maybe used to form training datasets (operation 620). The training datasetmay comprise paired datasets and un-paired datasets. The paired datasetsmay comprise co-registered MR image data acquired by different scannersor are of different styles. The paired datasets may include a referenceimage of a target style and an original image. The reference image of atarget style can be MR image taken by a selected scanner (provided by aselected vendor). The reference image of a target style may beconsistent with a standard form. The reference image may be synthesizedimage that is generated from raw image data by transforming it to astandard style. As described elsewhere herein, a target style mayspecify one or more characteristics selected from the group consistingof contrast, resolution, image distortion, skewness, color, size orother items.

The training step 630 may comprise a deep learning algorithm consistentwith the disclosure herein. The deep learning algorithm may be aconvolutional neural network, for example. In some cases, the deeplearning algorithm may be a deep residual learning network. In somecases, the deep learning algorithm may use Cycle Generative AdversarialNetwork (Cycle-GAN) for training on un-paired datasets.

The network model may then be used for transforming a MR image of anystyle to a target style. The MR image may be acquired by one or morescanners that may be of the same or different types (operation 640). Inan optional step, a transformation mode may be determined (operation650). The transformation mode may define the target style/form and/orone or more characteristics (e.g., contrast, resolution, imagedistortion, skewness, color, size) of a target style/form. Atransformation mode may be determined automatically or manually. In somecases, a target style may be pre-determined or pre-selected andautomatically loaded to the system. Alternatively or in addition to, auser may be permitted to select a target style or specify one or morecharacteristics of a target style. For example, a user may input, via auser interface, a target style. The target style may be provided via anysuitable formats on a GUI, such as a selection from drop-down menu(e.g., standard style, a vendor list, etc), direct input in a searchfield (e.g., input name of a vendor) or via other suitable means such asvoice command and the like. Upon determination of a target style, thecorresponding network model may be selected and the correspondingtransformation is performed on the input MR image (operation 660). Insome cases, the network model may be retrieved from a database that isin communication with the system.

Although FIG. 6 shows a method in accordance with some embodiments aperson of ordinary skill in the art will recognize that there are manyadaptations for various embodiments. For example, the operations can beperformed in any order. Some of the operations may be precluded, some ofthe operations may be performed concurrently in one step, some of theoperations repeated, and some of the operations may comprise sub-stepsof other operations.

FIG. 7 shows a block diagram of an example of MR imaging standardizationsystem 700, in accordance with embodiments of the present disclosure.The MR imaging standardization system 700 may comprise a system 710 fortraining a deep learning network model and inference. The system 710 canbe the same as the system as described in FIG. 1 . The system 710 maycomprise multiple components, including but not limited to, a trainingmodule 702, an image transformation module 704, a transformation modeselection module 706 and a user interface module 708.

The training module 702 may be configured to collect and manage trainingdatasets. The training module 702 may comprise a deep learning algorithmsuch as convolutional neural network (CNN). The training module may beconfigured to implement the machine learning methods as described above.The training module may train a model off-line or off-site.Alternatively or additionally, the training module may use real-timedata as feedback to refine the model. One or more trained network modelsmay be stored in a database 720.

The image transformation module 704 may be configured to transformimages to a target style using a network model that is trained by thetraining module. The image transform module may take one or more k-spaceimages or MR image data from one or more scanners of the same ordifferent types as input, and output MR image data with the targetstyle. In some embodiments, the image transform module may be incommunication with the database 720 such that upon determining a targetstyle or transformation mode, a corresponding network model may beretrieved from the database 720.

The transformation mode selection module 706 may be operably coupled tothe image transformation module and/or the user interface module 708.The transformation mode selection module 706 may be configured todetermine a transformation mode. The transformation mode may define thetarget style/form and/or one or more characteristics of a targetstyle/form. A transformation mode may be determined automatically ormanually. In some cases, the transformation mode selection module 706may automatically load to the system a target style that ispre-determined or pre-selected. In some cases, the transformation modeselection module 706 may analyze the input the image data andautomatically determine a transformation mode or target style. Thetarget style or transformation mode may be determined based onpredetermined rule(s). For instance, the target style may be determinedbased on an optimal quality of the output image. For instance, when theinput image data comprise data collected from two different types ofscanners, the target style may be determined to be the same as the stylethat has a higher resolution or better contrast.

In some case, the transformation mode selection module 706 may allow auser to select a target style or one or more characteristics of a targetstyle. For example, a user may be permitted to select a target style orspecify one or more characteristics of a target style. In response todetermining a target style, the transformation mode selection module 706may notify the image transformation module 704 for obtaining thecorresponding network model. In some cases, the transformation modeselection module 706 may receive a user input indicating a desiredstandard form or target form (e.g., standard style, a given vendor'sstyle, image resolution, field of view, color, contrast, etc). Thetransformation mode selection module 706 may be operably coupled to theuser interface module 708 for receiving user input and outputting aselected transformation mode or target style.

The user interface module 708 may render a graphical user interface(GUI) 740 allowing a user to select a transformation mode, a targetstyle or one or more characteristics of a target style, viewinginformation related to image transformation settings and the like. TheGUI may show graphical elements that permit a user to view or accessinformation related to image standardization. A graphical user interfacecan have various interactive elements such as buttons, text boxes andthe like, which may allow a user to provide input commands or contentsby directly typing, clicking or dragging such interactive elements. Forexample, a user may input, via a user interface, a target style. Thetarget style may be provided via any suitable formats on a GUI, such asa selection from drop-down menu (e.g., standard style, transformationmode, a vendor list, etc), direct input in a search field (e.g., inputname of a vendor) or via other suitable means such as voice command andthe like.

In some cases, the graphical user interface (GUI) or user interface maybe provided on a display 735. The display may or may not be atouchscreen. The display may be a light-emitting diode (LED) screen,organic light-emitting diode (OLED) screen, liquid crystal display (LCD)screen, plasma screen, or any other type of screen. The display may beconfigured to show a user interface (UI) or a graphical user interface(GUI) rendered through an application (e.g., via an applicationprogramming interface (API) executed on the local computer system or onthe cloud).

The imaging standardization system 700 may be implemented in software,hardware, firmware, embedded hardware, standalone hardware, applicationspecific-hardware, or any combination of these. The imagingstandardization system, modules, components, algorithms and techniquesmay include implementation in one or more computer programs that areexecutable and/or interpretable on a programmable system including atleast one programmable processor, which may be special or generalpurpose, coupled to receive data and instructions from, and to transmitdata and instructions to, a storage system, at least one input device,and at least one output device. These computer programs (also known asprograms, software, software applications, or code) may include machineinstructions for a programmable processor, and may be implemented in ahigh-level procedural and/or object-oriented programming language,and/or in assembly/machine language. As used herein, the terms“machine-readable medium” and “computer-readable medium” refer to anycomputer program product, apparatus, and/or device (such as magneticdiscs, optical disks, memory, or Programmable Logic Devices (PLDs)) usedto provide machine instructions and/or data to a programmable processor.The imaging standardization system can be a standalone system that isseparate from the MR imaging system. Alternatively or in addition to,the imaging standardization system can be integral to the MR imagingsystem such as a component of a controller of the MR imaging system.

The imaging standardization system 700 may comprise computer systems forimplementing the system 710 and database systems 720. The computersystem can comprise a laptop computer, a desktop computer, a centralserver, distributed computing system, etc. The processor may be ahardware processor such as a central processing unit (CPU), a graphicprocessing unit (GPU), a general-purpose processing unit, which can be asingle core or multi core processor, a plurality of processors forparallel processing, in the form of fine-grained spatial architecturessuch as a field programmable gate array (FPGA), an application-specificintegrated circuit (ASIC), and/or one or more Advanced RISC Machine(ARM) processors. The processor can be any suitable integrated circuits,such as computing platforms or microprocessors, logic devices and thelike. Although the disclosure is described with reference to aprocessor, other types of integrated circuits and logic devices are alsoapplicable. The processors or machines may not be limited by the dataoperation capabilities. The processors or machines may perform 512 bit,256 bit, 128 bit, 64 bit, 32 bit, or 16 bit data operations.

The imaging standardization system 700 may comprise one or moredatabases. The one or more databases 720 may utilize any suitabledatabase techniques. For instance, structured query language (SQL) or“NoSQL” database may be utilized for storing MR image data, raw imagedata, synthesized reference image data, training datasets, trainedmodel, target style, characteristics of a style or form, etc. Some ofthe databases may be implemented using various standard data-structures,such as an array, hash, (linked) list, struct, structured text file(e.g., XML), table, JSON, NOSQL and/or the like. Such data-structuresmay be stored in memory and/or in (structured) files. In anotheralternative, an object-oriented database may be used. Object databasescan include a number of object collections that are grouped and/orlinked together by common attributes; they may be related to otherobject collections by some common attributes. Object-oriented databasesperform similarly to relational databases with the exception thatobjects are not just pieces of data but may have other types offunctionality encapsulated within a given object. If the database of thepresent disclosure is implemented as a data-structure, the use of thedatabase of the present disclosure may be integrated into anothercomponent such as the component of the present invention. Also, thedatabase may be implemented as a mix of data structures, objects, andrelational structures. Databases may be consolidated and/or distributedin variations through standard data processing techniques. Portions ofdatabases, e.g., tables, may be exported and/or imported and thusdecentralized and/or integrated.

The network 730 may establish connections among various components in aMRI system and a connection of the imaging standardization system toexternal systems (e.g., databases, servers, MRI systems, etc). Thenetwork 730 may comprise any combination of local area and/or wide areanetworks using both wireless and/or wired communication systems. Forexample, the network 730 may include the Internet, as well as mobiletelephone networks. In one embodiment, the network 730 uses standardcommunications technologies and/or protocols. Hence, the network 730 mayinclude links using technologies such as Ethernet, 802.11, worldwideinteroperability for microwave access (WiMAX), 2G/3G/4G mobilecommunications protocols, asynchronous transfer mode (ATM), InfiniBand,PCI Express Advanced Switching, etc. Other networking protocols used onthe network 730 can include multiprotocol label switching (MPLS), thetransmission control protocol/Internet protocol (TCP/IP), the UserDatagram Protocol (UDP), the hypertext transport protocol (HTTP), thesimple mail transfer protocol (SMTP), the file transfer protocol (FTP),and the like. The data exchanged over the network can be representedusing technologies and/or formats including image data in binary form(e.g., Portable Networks Graphics (PNG)), the hypertext markup language(HTML), the extensible markup language (XML), etc. In addition, all orsome of links can be encrypted using conventional encryptiontechnologies such as secure sockets layers (SSL), transport layersecurity (TLS), Internet Protocol security (IPsec), etc. In anotherembodiment, the entities on the network can use custom and/or dedicateddata communications technologies instead of, or in addition to, the onesdescribed above.

Whenever the term “at least,” “greater than,” or “greater than or equalto” precedes the first numerical value in a series of two or morenumerical values, the term “at least,” “greater than” or “greater thanor equal to” applies to each of the numerical values in that series ofnumerical values. For example, greater than or equal to 1, 2, or 3 isequivalent to greater than or equal to 1, greater than or equal to 2, orgreater than or equal to 3.

Whenever the term “no more than,” “less than,” or “less than or equalto” precedes the first numerical value in a series of two or morenumerical values, the term “no more than,” “less than,” or “less than orequal to” applies to each of the numerical values in that series ofnumerical values. For example, less than or equal to 3, 2, or 1 isequivalent to less than or equal to 3, less than or equal to 2, or lessthan or equal to 1.

As used herein A and/or B encompasses one or more of A or B, andcombinations thereof such as A and B. It will be understood thatalthough the terms “first,” “second,” “third” etc. are used herein todescribe various elements, components, regions and/or sections, theseelements, components, regions and/or sections should not be limited bythese terms. These terms are merely used to distinguish one element,component, region or section from another element, component, region orsection. Thus, a first element, component, region or section discussedherein could be termed a second element, component, region or sectionwithout departing from the teachings of the present invention.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” or “includes” and/or “including,” when used in thisspecification, specify the presence of stated features, regions,integers, steps, operations, elements and/or components, but do notpreclude the presence or addition of one or more other features,regions, integers, steps, operations, elements, components and/or groupsthereof.

Reference throughout this specification to “some embodiments,” or “anembodiment,” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. Thus, the appearances of the phrase “in someembodiment,” or “in an embodiment,” in various places throughout thisspecification are not necessarily all referring to the same embodiment.Furthermore, the particular features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. It is not intendedthat the invention be limited by the specific examples provided withinthe specification. While the invention has been described with referenceto the aforementioned specification, the descriptions and illustrationsof the embodiments herein are not meant to be construed in a limitingsense. Numerous variations, changes, and substitutions will now occur tothose skilled in the art without departing from the invention.Furthermore, it shall be understood that all aspects of the inventionare not limited to the specific depictions, configurations or relativeproportions set forth herein which depend upon a variety of conditionsand variables. It should be understood that various alternatives to theembodiments of the invention described herein may be employed inpracticing the invention. It is therefore contemplated that theinvention shall also cover any such alternatives, modifications,variations or equivalents. It is intended that the following claimsdefine the scope of the invention and that methods and structures withinthe scope of these claims and their equivalents be covered thereby.

What is claimed is:
 1. A computer-implemented method for transformingmagnetic resonance (MR) imaging across multiple vendors comprising: (a)during a training phase, obtaining a training dataset, wherein thetraining dataset comprises a paired dataset and an un-paired dataset,wherein the training dataset comprises image data acquired using two ormore MR imaging devices provided by two or more different vendors, andwherein the image data have different image styles of MR imaging; (b)training a deep network model to learn a function that transforms imagesof different image styles to a target image style, wherein the trainingcomprises using a combination of supervised learning and unsupervisedlearning, and wherein the supervised learning comprises using the paireddata set of the training dataset to train the deep network model and theunsupervised learning comprises using the un-paired dataset to train thedeep network model; (c) after the deep network model is trained, usingthe trained deep network model to generate a synthesized MR image byprocessing an input MR image, wherein the synthesized MR image has thetarget image style and the input MR image has an image style differentfrom the target image style; and (d) displaying the synthesized MR imagewith the target image style on a display device.
 2. Thecomputer-implemented method of claim 1, wherein the image data acquiredusing the two or more MR imaging devices save different image styles. 3.The computer-implemented method of claim 1, wherein the plurality ofdifferent image styles are different in at least one of contrast,resolution and image distortion.
 4. The computer-implemented method ofclaim 1, wherein the target image style is pre-determined or selectedbased on a set of pre-determined rules.
 5. The computer-implementedmethod of claim 4, wherein the target image style comprises one or morecharacteristics including contrast, resolution or image distortion. 6.The computer-implemented method of claim 1, wherein the target imagestyle is consistent with an image style corresponding to a given MRimaging device, wherein the given MR imaging device is different from anMR imaging device used for acquiring the input MR image.
 7. Thecomputer-implemented method of claim 1, wherein the paired datasetcomprises reference image data acquired using a first MR imaging deviceand an original image data acquired using a second MR imaging devicethat is different from the first MR imaging device.
 8. Thecomputer-implemented method of claim 1, wherein the combination ofsupervised learning and unsupervised learning comprises training thedeep network model using the supervised learning and further enhancingthe deep network model using the unsupervised learning.
 9. Thecomputer-implemented method of claim 1, wherein the unsupervisedlearning comprises using Cycle Generative Adversarial Network.
 10. Thecomputer-implemented method of claim 1, wherein the deep learning modelcomprises U-net neural network structures.
 11. The computer-implementedmethod of claim 1, wherein the target image style is determined by auser via a graphical user interface.
 12. A non-transitorycomputer-readable storage medium including instructions that, whenexecuted by one or more processors, cause the one or more processors toperform operations comprising: (a) during a training phase, obtaining atraining dataset, wherein the training dataset comprises a paireddataset and an un-paired dataset, wherein the training dataset comprisesimage data acquired using two or more MR imaging devices provided by twoor more different vendors, and wherein the image data comprisesdifferent image styles of MR imaging; (b) training a deep network modelto learn a function that transforms images of different image styles toa target image style, wherein the training comprises using a combinationof supervised learning and unsupervised learning, and wherein thesupervised learning comprises using the paired data set of the trainingdataset and the unsupervised learning comprises using the un-paireddataset to train the deep network model; (c) after the deep networkmodel is trained, using the trained deep network model to generate asynthesized MR image by processing an input MR image, wherein thesynthesized MR image has the target image style and the input MR imagehas an image style different from the target image style; and (d)displaying the synthesized MR image with the target image style on adisplay device.
 13. The non-transitory computer-readable storage mediumof claim 12, wherein the image data acquired using the two or more MRimaging devices have different image styles.
 14. The non-transitorycomputer-readable storage medium of claim 12, wherein the plurality ofdifferent image styles are different in at least one of contrast,resolution and image distortion.
 15. The non-transitorycomputer-readable storage medium of claim 12, wherein the target imagestyle is pre-determined or selected based on a set of pre-determinedrules.
 16. The non-transitory computer-readable storage medium of claim15, wherein the target image style comprises one or more characteristicsincluding contrast, resolution or image distortion.
 17. Thenon-transitory computer-readable storage medium of claim 12, wherein thetarget image style corresponds to an image style of a given MR imagingdevice, wherein the given MR imaging device is different from a MRimaging device used for acquiring the input MR image.
 18. Thenon-transitory computer-readable storage medium of claim 12, wherein thepaired dataset comprises reference image data acquired using a first MRimaging device and an original image data acquired using a second MRimaging device that is different from the first MR imaging device. 19.The non-transitory computer-readable storage medium of claim 12, whereinthe combination of supervised learning and unsupervised learningcomprises training the deep network model using the supervised learningand further enhancing the deep network model using the unsupervisedlearning.
 20. The non-transitory computer-readable storage medium ofclaim 12, wherein the unsupervised learning comprises using CycleGenerative Adversarial Network.